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ABSTRACT 

This seven-part guide provides a framework for use in 
planning, developing, and implementing an evaluation for a tutoring 
program. Part I provides a brief introduction to the value of 
evaluating a tutoring program. Part II advises readers first to 
determine the purpose of their evaluation and the kinds of 
information to be examined, to consider their audience, and to 
specify the goals of the evaluation. Three phases of the evaluation 
process are described and relevant questions to be answered at each 
phase are posed. Part III describes how to conduct a qualitative 
evaluation for a tutoring program. A project evaluation matrix is 
included that poses evaluation questions, explains the documentation 
needed for each question, and lists the person responsible and the 
date of the documentation. Part IV discusses how to conduct a 
quantitative evaluation for a tutoring program, and includes sections 
on choosing the information to examine, selecting the evaluation 
design, choosing the evaluation participants, and collecting and 
analyzing the data. Part V focuses on reporting the evaluation 
findings through both narrative and visual information. Part VI 
addresses the issue of planning for future evaluation. Part VII 
presents a conclusion and includes a table that summarizes the 
essential components of a tutoring program evaluation. A number of 
relevant forms and materials are appended. (NB) 
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Preface 



Although we could see it with our eyes and feel it in our bones that the Adopt-a- 
Cub cross-age tutoring project was a success for both students and tutors, we soon 
found that we needed to be able to prove to potential supporters-both moral and 
financial-that this was really the case. Where and how to start? We were lucky. We 
had the personal interest of the National Dropout Prevention Center, the Colleges of 
Education and Liberal Arts, and Visions for Youth at Clemson University who 
recognized the potential in our project, and had the expertise and financial support to 
assist in this seemingly overwhelming task. With their guidance, an evaluation of our 
program was conducted. The results provided hard data in addition to the anecdotal 
information we had compiled from our tutors and supervisors to give us the boost we 
needed to get community support. As a result of this evaluation, we were able to obtain 
additional funding from our community's organizations and the state of South 
Carolina. We even received further recognition by the White House in May of 1992 
as the 765th Point of Light 

As a result of this experience, it was evident that there was a ^^reat need for an 
evaluation guide for the many tutoring programs springing up all over this country. 
Not only would such a guide give program coordinators the advantage of professional 
expertise that Adopt-a-Cub had in obtaining both financial and in-kind support, but 
also it would help them design a better program. Because evaluations make you aware 
of what is right and what is wrong with your program, you can then enhance the good 
parts and correct the bad parts. 

This guidebook is a concrete result of experiences with the evaluation processes 
used in Adopt-a-Cub and other tutoring programs. It contains everything you need to 
know about such an evaluation and provides you with all the checklists and fonns that 
you will need. We know you will find it an invaluable tool as you begin or continue 
your tutoring project 

Bill Wilson, Coordinator 
Adopt-a-Cub Program 
D.W. Daniel High School 

Marty Duckenfield 

National Dropout Prevention Center 

November 4, 1992 
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PARTI 



Introduction: Why Evaluate? 



If you are about to implement a tutoring projec you began a program last year 
and are considering expanding it, you would be wise to plan to evaluate your program. 
Why is it wise to evaluate a tutoring program? 

Evaluation of a tutoring program will help in planning, implementing, and developing 
the program. It provides an opportunity to examme both the strengths and weaknesses 
of the tutoring program and provides valuable feedback for improving the services 
provided. Decisionmakers need to know if the program objectives have been met or 
to what degree program objectives have been met before deciding if the program 
should continue or needs to be changed. Decisions may be based on both formal and 
informal evaluation of the pro^^.am. 

Many tutoring programs rely on outside funding and have a pressing need to 
demonstrate that the program is beneficial and is achieving its objectives. Additional 
funds may become available after an evaluation proves the program is effective in 
meeting its objectives. Evaluating a tutoring program will help answer questions like 
the following: 

• Is this tutoring program effective? 

• Can this tutoring program be improved? How? 

• How does this tutoring program help the community? 

The purpose of this guide is to provide a frameworic for project staff in planning, 
developing and implementing a useful evaluation for a tutoring program. Although 
this guide is not developed for use by an external evaluator, it may be helpful in 
bridging any communication gap between project staff and an outside evaluator of the 
project. 
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PART II 



Overview 



Before actually beginning, detenmine the purpose of the evaluation which will 
determine the kinds of infonnation that need to be examined. In addition, the audience 
that will be reviewing the results must be considered. After determining the purpose 
and the audience, specify the goals of evaluation. The goal(s) of the evaluation of a 
tutoring progiam might be to answer one or more of the following questions: 

• Is the tutoring program achieving its goals and objectives? 

• How can the program improve its services? 

• What lessons have been learned by implementing this program? 

• How can it be proven that the program is beneficial and worthy 
of funding? 

The question(s) to be answered by the evaluation will determine if the evaluation 
design will be qualitative, quantitative, or a combination of the two. Qualitative 
evaluation attempts to understand how program participants are thinking and feeling 
so that a perspective of the program from the user's viewpoint can be obtained. 
Qualitative evaluation is especially helpful in answering questions about how to 
improve services and what lessons were learned. Qualitative data can also provide 
very powerful answers conceming the benefits, worthiness for funding, and how 
well the program is achieving its goals and objectives. 

Quantitative evaluation emphasizes numerical facts about the effectiveness of the 
program in reaching its goals and objectives. Funders are often very definite about 
their desire to examine what they refer to as the "bottom line'' conceming a program. 
What is the cost/benefit ratio for participants? This information is best obtained 
through quantitative methods. 

A combination of qualitative and quantitative methods would he best for evaluating 
most tutoring programs. For either qualitative or quantitative evaluation, several 
essential steps must be taken before the evaluation can be conducted. The first step 
is to design a framework for managing the evaluation. Identify the major phases of the 
evaluation and outline the specific steps within each phase that are important to the 
evaluation. An example follows. 



Phase 1 : Context and Baseline Assessment 

The first phase of the evaluation might collect information to describe the environment 
or context of the school and community just prior to the beginning of the tutoring 
program and to assess the need for the program. Baseline data might be gathered 
during phase I of the evaluation which includes answers to the following questions: 

• What is the skill level of students before the beginning of the 
tutoring program? 
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• What are the concerns of the teachers and parents about children*s 
achievement in school? 

• How many children are retained in each grade level? 

• How many children have below average grades? 

• How many discipline referrals were made in the previous year for 
each grade level? 

Phase 2: Program Implementation 

The second phase of the evaluation might focus on the implementation of the 
program. This phase of the evaluation is helpful in detennining how to improve the 
program. The following questions might be the focus of the second phase of the 
evaluation: ^ 

• What woiiced well and what didn*t work so well within the tutoring 
program? 

• What were the problems? 

• What response was made to the problems? 

• How many volunteers were involved as tutors? 

• How mu';h and what kind of training did the tutors receive? 

• What other kinds of training were needed? 

• How well did various par: of the program (for example, tutors, teachers, 
parents) communicate? 

• What important lessons were learned about tutoring programs? 

Phase 3: Impacts 

The third phase of the evaluation might focus on detennining the outcomes of the 
tutoring program for the students involved. This phase of the evaluation might gather 
infonnation to answer the following questions: 

• What effect did the program have on grades? 

• What effect did the program have on school attendance? 

• What effect did the program have on grade retention? 

• What effect did the program have on discipline referrals? 

• What effect did the program have on self-esteem? 

O /() 3 



• What effect did the program have on homework behavior? 

• What effect did the program have on children's goals? 



The above example illustrates why acombination of bothqualitative and quantitative 
methods for evaluating a tutoring program are helpful. Although a particular phase 
may seem to emphasize one type of data over another (for example, qualitative in phase 
2 and quantitative in phase 3), there is not a clear delineation of type of data for each 
phase. Both kinds of information are helpful to answer the questions in all three 
phases. 

Ideally, an objective outside evaluator would be employed to conduct the evaluation. 
Employing an outside evaluator can be an expensive undertaking, especially for 
programs that are working on limited or small budgets. However, there are other 
alternatives. Program staff can conduct a useftil evaluation if it is well-planned and 
well-executed. This manual should be helpful for program staff in conducting an 
evaluation of a tutoring program. 

One essential step that must be completed before collecting evaluation infomiation 
is to secure permission, through infomied consent, for participants to take part in the 
evaluation. Legally, pemiission must be obtained, especially if the participants will 
be administered any assessment instmments. Because of the 1974 Family Education 
Rights and Privacy Ac^ also called the Buckley Amendment, it is essential that anyone 
conduaing an evaluation which involves children first describe and explain the 
evaluation to parents. Permission from parents must be secured for students to 
participate in the evaluation. This permission slip should: 

• Use wording that is understandable to the parents. 

• Tell the parents exacOy what the evaluation involves by describing 
the following: 

a. purpose, 

b. procedures, 

c. any risks to loss of confidentiality, 

d. safeguards to be used to protect the child's anonymity and 
confidentiality, and 

e. expected duration of the child's participation. 

• Include an invitation for the parents to ask any questions at any time about 
the evaluation and its procedures. 

• Tell the parents that participation is voluntary and that they may refuse to 
participate. Further, state that upon participating, parents may withdraw, or 
be withdrawn, from the evaluation at any time without jeopardizing the 
quality of the tutoring to be given or the interest the tutor has in the child. 



• Provide a statement that grants permission for the child to participate in the 
evaluation. 

• State that the information obtained will only be used for the purpose of 
evaluating the program. 

• Include a signature block for each of the following: 

a. parent, 

b. witness, and 

c. evaluator. 

Include one date block for the group of signatures. Include an initial block 
at the bottom of each page for the parent's initials (except for the last page 
where the signature(s) arc obtained). In the case of minors, both parents or 
an appropriate guardian must sign and the relationship to the minor must be 
stated underneath the signature. If both parents' signatures are not available, 
an explanatory statement must be included. It is optional a5. to whether or 
not a minor over the age of seven shall be asked to sign. 

An -example of a consent form for use in a tutoring program is included in the 
Appendix. The form may be copied and used as it appears in the Appendix or it may 
be adapted as needed for a particular program. 

Since the Buckley Amendment also stresses privacy, evaluators must design and 
implement ways to code all information about participants. Names should be removed 
immediately from documents and replaced with the code assigned to each participant 
so that individual privacy can be assured. A master list of codes assigned to 
participants should be kept in a secure place which is accessible to only one or two 
people to minimize the risk of loss of confidentiality. 
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PART III 



How to Conduct a Qualitative 
Evaluation for a Tutoring Program 

The following list identifies some of the materials and tools needed to conduct a 
qualitative evaluation of a tutoring program. Some evaluations will require more 
materials and tools than others, depending upon the particular evaluation needs of the 
program: 

• notebook or index cards for observation notes, 

• camera, 

• tape recorder, 

• a camcorder and VHS machine, and 

• Evaluation Guide for Tutoring Programs. 

Determine what major questions arc to be answered by the evaluation. The major 
questions for a qualitative evaluation should iielp identify the lessons that were learned 
about planning and conducting this particular program. These major questions will 
probably be related to the program goals and objectives; however, for a qualitative 
evaluation, there may not be a one-to-one correspondence with program goals and 
objectives. Once the major questions are identified, refining the major questions with 
subqucstions is useful. After all the major questions and subqucstions arc identified 
and refined, assign each question to the appropriate phase of the evaluation. Major 
questions and subqucstions for a tutoring program evaluation might include those in 
the following example: 



Example: 

L What M^as learned about identifying children who can benefit from tutoring? 

a. How were children identified for the program? 

b. Which children benefited most from the program? 

c. Which children benefited least from the program? 

d. Were any children overlooked? 

e. Did the method of identification affect participation? In what ways? 

2. What was learned about enablers and barriers to program implementation? 

a. What events, people, policies, procedures enabled implementation 
of the program? 

b. In what way did each enabler identified above facilitate program 
implementation? 

c. What events, people, policies, procedures created barriers to program 
implementation? 

d. In what way did each barrier interfere with program implementation? 

e. What response was made to each barrier? 
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3. What was learned about the effectiveness of the program components? 

a. What were the program components? 

b. How did each program component operate? 

c. Were any additional components added to the program? 

d. What effect did the changes have? 

e. Could any of the program components be eliminated with little or no 
effect on the children? 

/ What effect did the staffing pattern have on the program' s success? 
g. What effect did staff training have on the program's success? 
e. How were the children transported? 

4. What was learned about the intended and unintended impacts on children?*" 

a. What impact did the program have on goal setting? 

b. What impact did the program have on grades? 

c. What impact did the program have on retention? 

d. What impact did the program have on discipline referrals? 

e. What impact did the program have on homework behavior? 
f What impact did the program have on classroom behavior? 

g. What impact did the program have on school attendance? 

h. Were any unintended positive impacts identified? 

i. Were there any unintended negative impacts identified? 



♦Quantitative information (for example, change in grades) is often embedded in or 
needed to answer a qualitative question. 

The following matrix suggests additional questions and shows how each question 
could be answered, by whom, and by when. 
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How TO Conduct a Qualitative Evaluation for a Tutoring Program 



Project Evaluation Matrix* 



Evaluation Question 



1 . Have pre-projecl benchmarks been 
established? 

a. skill level of children 

b. concerns of teachers and parents 
about children's school 
achievement 

c. grade retention 

d. grade point averages 

e. discipline referrals 

f. self-esteem 

g. goal seuing 

2. Have project goals and objectives 
been detennined? 

3. Have strategies been developed to 
meet project goals and objectives 

4. Has a Project Advisory Committee 
been formed? 

5. Has the Advisory Committee 
reviewed project goals, objectives, 
& strategies? 

6. Has a calendar been developed? 

7. Is there a staffing plan? 



8. Have initial participants been 
identified? 

9. Have project staff (volunteers) 
been identified? 

10. Have families been supported 
to work with c aldren at home? 

1 1 . How much and what kind of 
training did tutors receive? 



12. Were training materials 

a. appealing? 

b. useful? 

c. effective? 



Documentation 
Needed 

Tests results. 
Survey results, 
Grade retention records. 
Grade point averages. 
Discipline referrals 
Self-esteem ratings 
Lists of children's goals 



Written goals and objectives 



Work scope for project 
Description of strategies 

List of Advisory Committee 
members 

Revised work scope, 
minutes, results of 
recommendations 

Calendar of project events 

Job descriptions, list of 
personnel, and assignments 

List of initial participants 



List of volunteers and 
signed contracts 

Record of home visits, list of 
information provided to families 

Description of staff training 
content and list of dates 
training received 

Copies of materials evaluations 
Assmnt. of attractiveness, etc. 
Assmnt. of helpfulness, etc. 
Assmnt. of changes 



Person 
Responsible 

School Counselor 
Project Evalualor 
Teachers & Parents 



Date of 

Documentation 
Julyl 



Project Director 
Project Director 
Project Director 

Project Director 

Project Director 
Project Director 

Project Director 

Project Director 

Project Director 

Project Director 

Project Evalualor 



August 1 
Augus: 15 
August 15 

September 1 

September 15 
September 15 

October 

October 

October 

October 

October 
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How TO Conduct a Qualitative Evaluation for a Tutoring Program 



Project Evaluation Matrix 



Evaluation Question 



13. What other training was needed? 

14. Has curriculum for tutoring 
been developed? 

15. Were tutoring materials 

a. appealing? 

b. useful? 

c. effective? 

1 6. Have tutoring program impacts 
been evaluated? 

a. skill level of children 

b. concerns of teachers and parents 
about children *s school 
achievement 

c. grade retention 

d. grade point averages 

e. discipline referrals 
self-esteem 
goal setting 



f. 



Documentation 
Needed 

Description of problems 
Copy of tutoring curriculum 



Assmnl. of attractiveness, etc. 
Assmnt. of helpfulness, etc. 
Assmnl. of changes 

Attendance logs» tests results, 
survey results, grade reten- 
tion records* grade point 
averages, discipline referrals, 
self-esteem ratings, lists of 
children *s goals, assessment 
of change in homework 
behavior and in-home reading 
practice, assessment of change 
in library card holders & users, 
assessment of change in student 
and parent engagement, 
assessment of change in student 
achievement 



Person 
Responsible 

Project Evaluator 

Project Director 

Project Evaluator 



Project Director 
Project Evaluator 
School Counselor 
Project Evaluator 
Teachers & Parents 



Date of 

Documentation 

November, February 
November 

November 



January 
March 
July 1 



17. What worked well and what 
didn't work so well within the 
tutoring program? 



Notes from interviews, focus 
groups, and observations 



18. What response was made to problems? Description Ot responses 



1 9 . How did various parts of the program 
communicate? 



20. What important lessons were learned 
about tutoring programs? 



2 1 . Were project results disseminated? 



Copy of communications 
Notes from interviews, focus 
groups, observations, 
attitude survey of parent 

Description 



Dissemination plan & 
schedule 



Project Evaluator 



Project Director 



Project Director 
Project Evaluator 



Project Evaluator 



Project Director 
Project Evaluator 



December 

February 

June 

January 

March 

July 

December 

February 

June 



December 

March 

July 



July 



*This project evaluation matrix is adapted from work designed by Greg R. 
College of Education, Clemson University. 



Wei sens tein. Assistant Dean of Research, 
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How TO Conduct a Qualitative Evaluation for a Tutoring Prograra 



After all major questions and subquestions have been identified and assigned to the 
appropriate phase of the evaluation, their importance needs to be prioritized to assist 
with assigning resources and determining if completing the full evaluation Is possible. 
When determining the importance of the questions to an evaluation, it is also important 
to consider the audience that will be reading the final evaluation report. For example, 
different kinds of infonnation about the lessons learned will be needed by the 
community, by educators, and by funding sources. Each audience must be considered. 

Once the questions have been written, the management plan for the program 
evaluation can be effectively developed to ensure that all of the questions will be 
addressed. Determining the evaluation questions and the priority of those questions 
will guide the decisions about what data will be collected, what aspects of the program 
will be observed, who will be interviewed, when and how the observations and/or 
interviews will be conducted, what will be asked or observed, and when and how otlier 
data will be collected. 



Four elements must be part of the qualitative evaluation. First, the qualitative 
evaluator must be close enough to the participants and to the program being 
evaluated to understand the depth and details of what goes on in the program. 
Second, the evaluator must aim to describe what actually takes place and what 
people actually say. Third, pure description of participants, activities, and 
interactions is provided. Fourth, direct quotations from participants, both what 
they speak and what they write, are recorded as data (Lofland, 1971). 



Observations. Information for a qualitative evaluation is obtained primarily from 
interviews and observations. Observations may be random or systematic, formal or 
informal,or acombination. The greatest understanding of the program may be gained 
by designing a combination of random and systematic, forma! and informal 
observations. Formal observing occurs when both the participant and the observer 
know ahead of time that ihc observation will take place. Furthermore, the observation 
may be focused on capturing adcscription ofone or two targeted aspects of the tutoring 
program, for example, rather than being conducted as an open-ended observation. 
Informal observation may occur at any lime or place within the program. It may be 
conducted by a nonparticipating evaluator or by an observer who is also participating 
in tlie program as either a tutor or as a tutee. Informal observation may cither focus 
on a targeted aspect of the program or be open-ended. 

There are many reasons why observation is useful. Observation during tutoring 
will assist in making an initial assessment of the child *s abilities; determining a child's 
areas of strength and which areas need strengthening; making individual plans for 
future tutoring sessions based on observed needs; conducting an ongoing check on the 
child's progress; learning more about the child's development in particular skill areas; 
resolving a particular problem involving the child during the tutoring session; 
reporting to parents or specialists in health, speech, mental health; and in gathering 
information for the child *s folder for use in ongoing guidance and placement of the 
child in school (adapted from Bcaty, 1990). 

ERIC ° i ' 



How TO Conduct a Qualitative Evaluation for a Tutoring Program 



Methods for recording the information from observations vary. Observations may 
be electronically recorded using a tape recorder or video camera or manually recorded 
as anecdotal records or as running records. Electronically recorded observations are 
usually transcribed for later analysis. Anecdotal records are descriptions written some 
time after the tutoring session has occurred. Running records are notes taken to record 
as much as possible about an event while it is happening. 

The following general guidelines will assist the observer in making and recording 
useful observations of the tutoring program, 

• Make a clear distinction between what is actually seen or heard and 
conclusions. 

• Try to interpret what is observed from the child's viewpoint rather than 
imposing adult perceptions. 

• Draw no conclusions which cannot be positively justified by the actual 
observation. 

• Describe, in writing, the purpose and procedure involved in the observation. 

• Adopt a low profile by selecting an unobtrusive vantage point for observing 
and by sitting rather than standing. Get the tutor's advice on where to sit. 

• Information gathered on individuals should be treated confidentially and 
with respect, 

• Take down exact words whenever possible and note exact behavior 
(i,e., body language). 

• Be as objective as possible. Avoid letting personal feelings about the child 
slant or bias the observation. 

The identity of observer and the code name or description and any special 
characteristics (age of siblings, disability, etc.) of the tutec should be on every sheet 
used for observation. More specific suggestions for anecdotal records and for running 
records and examples of each are in ihc following sections. 



Anecdotal Records 

1, Write down the anecdote as soon as possible after it occurs. 

2. Identify the basic action of the key person and what was said. 

3, Include a statement that identifies the setting, time of day, and basic activity. 

4. In describing the central character's actions or verbalizations, include the 
responses or reactions of other people in the situation. 
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How TO Conduct a Qualitative Evaluation for a Tutoring Program 



5. Whenever possible note exact words used to preserve the precise flavor of 
the conversation. 

6. Preserve the sequence of the episode. 

7. Three levels of action should be included. 

a. the major behavior 

b. the smaller parts of the action within the larger action 

c. how the main action is carried out 

8. Be objective, accurate, and complete. 
Example: 

An afterschool tutoring program at the Girls and Boys Club involves a 
variety of educational programs presented when the children arrive. The 
following occurred during one meeting. 

Six children sat around a table while a volunteer from the Drug and Alcohol 
Abuse Prevention Program presented a program. During the program the 
children answered various questions and often volunteered information. One 
nine-year-old child stated that someone had offered some pills to him, telling 
him it was candy. He refused to take them. He continued to describe a drug 
sale which occurred just outside his home. None of the other children 
volunteered firsthand information about drugs. 

One six year old had learned what to do if someone came to him, told him 
his mother had been injured and to get in the car so he could go to her. He said 
that person would have to tell him the code word or he wouldn't get into the car. 
He had apparently learned this through a special program at his school. 

Following this presentation, the children went to the next room to play 
educational board games. Two adult volunteers were seated at the table with 
the children. One of the children, who played with an adult observer, was very 
pleased to win 2 out of 3 games of Connect Four. 

(Note: The tutoring session followed the game time.) 

Running Records 

1 . Describe the scene as it is when the observer begins the running notes. 

2. Focus on the subject(s) behavior and whatever in the situation itself affects 
this behavior. Two cases in which events or conditions removed from the 
subject need to be considered: 

a. an action or circumstance that would normally impinge on the subject 
but does not do so in this case; 
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b. an action or circumstance that leads to a change in the subject's 
situation, even though the subject is not initially aware of the 
change. 

3. Be as accurate and complete as you can about what the subject says, does, 
and responds to within the situation. 

4. Put brackets around all interpretive material generated by the observer so 
that the description itself stands out clearly and completely. 

5. Include the "how" for whatever the subject does. 

6. Give the "how" for everything done by anyone interacting with the subject. 

7. For every action, report all the main steps in their proper order. 

8. Describe behavior positively, rather than in terms of what was NOT done. 

9. Put no more than one main behavior in a sentence. 

10. Put no more than one thing done by a person other than the subject into a 
single sentence. 

11. Do not report observations in terms of the time an event happened, but do 
marie off predetermined time intervals (one-minute or two-minute 
intervals, for example). 

1 2. Write in everyday language. 

13. Use observational tools whenever possible (tape recorders, cameras, or 
videotape) and transcribe notes. Time, purpose, and budget will 
determine what, if any, observational tools you will be able to use and 
what steps you will take in preparing a final copy of your observations. 



Example- 
Observation of Tutoring Program at the Community Recreation Center, Most 
of the participants are in the 3rd grade. 

3:30 p.m. A group of 5 children and 2 adults are seated in three groups in a 
kitchen wth tables and chairs arranged in raws. 

• Twofemale children are with one female adult. One female 
child is reading to the adult, the other female child is reading a 
book while standing beside the female child who is r^.ading 
aloud, 

• A female is seated beside another female adult. The child is 
cutting out a picture. 
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• One male child is sitting beside a female child at the end of the 
table. 

3:32 Male: "You can't do your spelling because you forgot it.*' 

(Gets up and walks around.) 

Female: (Working on science homework at the same table) "My 
mama can call her and get it" 

Male: (addressed to group of three) "May I borrow some 
paper?" (Gets up.) 

Female: (4 spaces away) "Here." 

Male: "May I go get some water?" (Gets up and leaves room.) 

3:34 Female: Walks up behind adult who is listening to children read 

with a smile for a hug. 

All other children are continuing to work. 

A new female child walks into room and sits beside female adult 
(with child cutting out picture). 

One female child asks, "Did I pass my test?" 

Adult answers, "You missed two." 

3:36 Three female children go get journals. One child, however, must 

leave when parent arrives before she is able to sit and write in her 
journal. 

3:38 Male: "I never get to play. I want to play pool." (Up and walking 

around.) 

Adult: "Those boys aren't going to let you play pool." 

Male: "If they're smacked in the mouth." (Sits down again.) 

Female: "Come on." 

Male: "I can't, Tm not through." 

Female: "Well, get through." Child leaves. 

Female 1 : "Are you going to stay here or go to the playground?" 

Female 2: "Vm going to the playground." 

Adult: When your friend (tutor) comes in 15 minutes, she'll come 
and get you. 

Female 2: "Bye." 
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Female 1: "Bye . See you next Thursday" (Small female to the 
other,) One child leaves for the playground. The other leaves 
either for another part of the building or to go home. 



Adult: "Lefs sound it out!' (Adult walks over, sits beside male 
child, and assists him in sounding out "wanted",) 

Male child continues to work on homework while adult sits beside 
him with arm resting across the back of chair in which male is 
sitting. 



[Evaluator's note: Although the male child had great difficulty in staying on 
task when only he and a female child were sitting at the table, when he was 
receiving one-on-one assistance from the female adult tutor, he worked with 
focus and interest.] 



[Evaluator's note: Both laugh and participate in repeated, relaxed, warm 
interactions. The male child made eye contact when talking about being left 
handed. Others in the family are right-handed.] 

The male child leaves for the playground. 

Other examples of qualitative methods of program observation include: 

• written reports by program staff, observers, or participants; 

• cvaluators' notes on interviews; 

• examples of participant work; 

• evaluaiors* notes describing participant work; 

• tape recordings of interactions; 

• videotapes of interactions; 

• notes derived from videotapes or audiotapes; 

• questionnaires, surveys, etc., returned by parents and/or participants; 

• participant journal writing; 

• staff journal writing; and 

• evaluators' notes from on-site observation of program. 

This UsU was adapted from EvaluaUns Educational Programs by Mitchell Lazarus. 



3:40 



Male: '7 need help. How do you spell 'wanted' ? 



3:42 



Adult: Takes papers and shakes off erasing "crumbs.** 

Male: "Tm left-handed. Everyone else in my family is right- 
handed.'' 
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The following exam pies illustrate the powerful meaning which can be communicated 
about a program through qualitative evaluation methods. The first two examples 
represent the viewpoint of high school tutors in a cross-age tutoring program. The next 
examples from written reports, verbal reports, and on-site observation represent the 
viewpoints of planners, coordinators, and evaluators of a variety of tutoring programs. 

Examples: 

"After meeting her family, I got a different perspective of my [childj. I 
understand the insolence she has toward school, why she might think everyone 
hates her, and why she can t stand up to her best friend, Donna. 

I've learned so many different ways to get a point across, whether it is 
school work, friendship - love. It takes patience to somehow explain the 
'unexplainablej to build a friendship, but somehow we have, together." 
(Excerpt from a tutor s journal.) 

"I used to think I would not be of help to Terri, but as the hour on Tuesdays 
goes by, I pick up on little things. 7 want to make an A or Bfor you,' she says. 
My next project is for Terri to say, 7 want to make an A or Bfor me!' But 
grades aren't all that is important. Terri needs the pride and security to believe 
she is worth something, that she has something to offer. It makes us both feel 
special to know that she looks forward to seeing me, and I look forward to 
seeing her. 

Of course, there are the stories she tells, the battles in family court, whether 
our tutoring place will be changed because she is moving to a new school, the 
father in jail, the new boyfriend living with the family. I was completely 
shocked by these things at first, but now I don't bat an eyelash when Terri tells 
me she jumped out of her second story bedroom window. We like each other, 
we help each other, we love each other," (Excerpt from a tutor's journal.) 

"One teacher commented there was a tremendous change after just one 
visit." (Statement from verbal report of program director.) 

"In one class, three children were identified to receive tutoring. The 
teacher was sure at that time that two of the three children would be retained. 
Now there is a 60% chance, according to that same teacher, that all three 
children will be promoted. 

All 30 of the children who participated in the summer program attended a 
basketball game at the High School. Twenty-three of the 30 had never been to 
an athletic event. I plan to have them attend the High School graduation this 
spring. There won' tbea transportation problem because the High School has 
adopted this elementary school." (Statement from verbal report of program 
planner.) 
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"David cant wait to get home and show his homework to his mom. The 
Homework Site makes him feel special. Now his older brother who previously 
was unwilling to have a tutor says he is willing to be tutored.** (Statement from 
interview with program coordinator.) 

"On some days last week there were more than one tutor per child because 
of the heavy participation from college volunteers last week. We hope to keep 
some of these students as regular tutors every week/' (Statement from County 
Youth Development Council member.) 

Interviews- Interviews may be conducted with individual participants or with a 
group of selected participants in a group discussion called a focus group. In addition, 
interviewing may be formal, with a predetermined set of questions which are asked 
at a particular time and place by a specified interviewer, or informal, more as 
conversations among equals while participating in the program. The infonmation is 
recorded as notes or with a lape recorder. 

Examples of interview questions: 
For the teacher — 

Why do you recommend students for the tutoring program? 

What kind of improvement, if any, have you seen in your students? 

How could the tutoring program be improved? 
For the child — 

How has the tutoring program helped you? 

What do you like the most about tutoring? 

What do you like the least about tutoring? 
For the parent — 

What kinds of changes, if any, have you seen in your child since sihe started 

attending tutoring? 
Does your child regularly talk about the tutor or the tutoring sessions? 
For the tutor — 

What kinds of things do you do with your student during a regular tutoring 
session? 

What kinds of things do you discuss/talk about with your student? 

What is the toughest thing about being a tutor? 

If you could change one thing about your student, what would it be? 
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An evaluator who comes from outside the regular program staff, program staff, or 
a combination of an outside evaluator (usually less available and more costly but also 
more objective) and program staff (more available and less costly but also less 
objective), may be used to conduct the interviews and the observations. Participants 
who will be interviewed and/or observed may be preselected to be representative of 
all participants, may be randomly selected, or a particular segment of participants may 
be targeted for better understanding to improve services which are provided to that 
particular group of participants. 

One question that is frequently raised about the use of qualitative methods is how 
the information gathered is analyzed. Qualitative ev aluators use a variety of approaches, 
one of which is to attempt to categorize responses and generate a report about the major 
categories found. Another approach is to write a report describing the observations 
of the evaluator. Essentially, qualitative evaluation usually produces a written 
narrative report while quantitative evaluation produces more numerical data. 



PART IV 



How to Conduct a Quantitative 
Evaluation for a Tutoring Program 

The following list identifies some of the materials and tools needed to conduct a 
quantitative evaluation of a tutoring program. Some evaluations will require more 
materials and tools than others, depending upon the particular evaluation needs of the 
program: 

• notebook for recording data notes, 

• computer with computer software or calculator, 

• a research or statistic book, 

• assessment instruments, 

• forms for gathering data, and 

• Evaluation Guide for Tutoring Programs, 

A tutoring program needs to have clearly defined objectives before quantitative 
evaluation goals can be determined. Thinking about the changes the program is 
hoping to make in the children is helpful when trying to write objectives for the 
program. Objectives should be clearly written and realistic. For example, the 
objective students presently failing all subjects will have A ^s in every class by th e end 
of the program would not be realistic because it would require a great amount of 
change beyond the capability of most children. A more realistic objective states at 
least 70 percent of the children who participate mil improve in one or more subjects 
by one letter grade. After objectives have been written for the program, it is much 
easier to write the evaluation goals and objectives. 



Choosing the Information to Examine 

Detemiining what kinds of information need to be examined will depend on the 
goals and objectives of the program being evaluated and on the evaluation goals. For 
a quantitative evaluation, it may be desirable to gather some specific information 
about participants in order to see if any generalizations can be drawn about certain 
groups of participants. For example, the following arc types of information which 
could be used to make comparisons among participants in a cross-age tutoring 
program: 

• sex. 



• age, 

• grade, 

• ethnicity, 

• eligibility for free or reduced lunch. 



ERLC 



19 



How TO Conduct a Quantitative Evaluation for a Tutoring Program 



• type of family stnicture, 

• school in which student is enrolled, 

• eligibility for special education, and 

• history of retention. 

Evaluators may wish to determine if there are any similarities between similar 
groups of participants. Infonnation that could be examined include . Tiber of days 
missed from school; number of days missed from tutoring; grades for all four marking 
periods (especially reading, math, and GPR); number of discipline referrals for all 
marking periods; pre and post ratings of self-esteem by parents, teachers, and 
participants; changes in standardized test scores; changes in behavior (homewoik, 
discipline, attitude) as noted by teachers; or changes in attitudes as measured by a :!ttitude 
assessment instrument. (See the Appendix for an example of how to compute a GPR.) 

The infonnation listed above varies in how sensitive it is to the effects of the 
program. Differences in grades, attendance, and discipline referrals are more sensitive 
measures of program effects resulting from a one-year tutoring effort than are 
standardized achievement test scores. 



Choosing the Evaluation Design 

If the evaluation is to be only quantitative, there are several different evaluation 
designs. Each of the designs normally considered for evaluation studies will be 
discussed, including those which are useful and those which are not useful for 
evaluating a tutoring program. The reasons why a particular design is not useful for 
evaluation of a tutoring program will be discussed. 

The first three designs discussed below utilize a control group. Often the realities 
of educational programs prevent the use of control groups. For example, a control 
group design poses an ethical dilemma for a proven, existing program. To choose not 
to serve a group of participants may not be ethical, especially if it is known that the 
participants will benefit from the services offered. However, for a pilot program, 
without proven effectiveness, the question of who will not receive services may not 
pose an ethical dilemma. In addition to the ethical reality, there may not be a treatment 
alternative available for a comparative study. 

Pretest-posttest Control Group 

The classic experimental design is the pretest-posttest control group design. It is 
highly acceptable as a design for evaluations because it provides the basis for 
comparing results, a requirement for meaningful interpretation. The use of a control 
or comparison group provides the best prospect of obtaining meaningful answers to 
evaluation questions (Wolf, 1990). This design involves having two groups of 
participants who are randomly selected. One group of participants receives the 
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services provided by the program while the control or comparison group receives no 
services or limited services. Random procedures arc also used to determine which 
group receives the services provided by the program. Randomization insures that 
the groups are equal at the beginning, 

Posttest Only Controi Group 

A second design often used by researchers, but not recommended for evaluation 
(Wolf, 1990), is the posttest only control group design. In order to use this design for 
evaluation, one must assume the learners knew nothing about what they were 
supposed to learn before the program began. By their very definition, this assumption 
cannot be made for tutoring programs. Therefore, the posttest only control group 
design is not recommended for tutoring programs. 

Nonequivalent Control Group 

Although the use of a control group and randomized selection is recommended 
and desirable for a quantitative evaluation, it is not absolutely necessary when 
conducting an evaluation of a tutoring program. It is acceptable to use other designs. 
For example, the nonequivalent contrast group design is identical to the pretest-posttest 
control group design in all respects except for the absence of random assignments of 
participants to groups. In order to use this design, the evaluator must perform some 
additional analysis to determine that the group used for comparison and the group 
receiving the services are equivalent. Pretest data and other descriptive information 
about the two groups such as age, sex, socioeconomic status, previous school per- 
formance, and scholastic ability will be needed to establish equivalency (Wolf, 1990). 

Single Group Pretest-posttest 

The single group pretest-posttest design involves identifying a group, pretesting its 
members, exposing them to the tutoring program, and testing them again. The dif- 
ference between the pretest and the posttest measures is intended to measure the effects 
of the tutoring program. However, there may be a number of alternative explanations, 
other than the effects of the tutoring program, which could explain the differences in 
pretest and posttest scores. In order to use this design, the ev aluator must use logic and 
judgement to systematically eliminate each competing alternative explanation. In 
order to use logic and judgement to systematically eliminate each competing alter- 
native explanation, the evaluator must have sufficient knowledge about the particular 
situation surrounding the tutoring program. The evaluator must spend a considerable 
amount of time in the situation, or have access to those who do, in order to acquire the 
necessary knowledge to deal with competing explanations (Wolf, 1990). 

Case Study 

The one-shot case study can be used for evaluating tryouts of a part of the tutoring 
program to get an initial estimate of how particular learning experiences or materials 
are working. It may also be used when it can be assumed that learners have very little 
or no knowledge when they enter the tutoring program, and it is not possible to have 
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a comparison group (Wolf, 1990). Since the chief requirements for use of this design 
are (a) novel and specialized subject matter and (b) learners without proficiency in the 
subject, it is not likely that it can be used for many tutoring programs. 

Interrupted Time-Series 

The interrupted time-series design is useful if the tutoring program is introduced 
to all students at once. Data are obtained on a regular basis, before the initiation of 
the program, during the program, and after the program has ended (if applicable). 
Existing records such as standardized test scores, average daily attendance rates, 
discipline referrals and grades are examples of information that could be gathered 
on a regular and routine basis and used to estimate the effects of the introduction 
of a tutoring program. 

Choosing the Participants and Obtaining 
Informed Consent 

Once the design has been selected, how the participants will be selected must be 
determined. If the group is small, then all panicipanLs may be selected. However, if 
the group is large and resources available for evaluation are small, then random 
selection of participants (usually 10-20 % of the total group) for data collection is 
usually the best selection process to use because it eliminates bias. 

If both a control and a treatment group are to be selected, then assignment to these 
two groups should also be random . One method to randomly assign participants to one 
of two groups is to put all of the names of participants in a hat, draw the name of a 
participant from the hat, record the name drawn, replace the participant's name in the 
hat, and continue drawing and recording each new name drawn until half of the 
participants have their names recorded. This procedure of replacing the participant's 
name in the hat even after it has been drawn and recorded insures that every participant 
has an equal opportunity of being drawn from the hat. 

After participants are selected, obtain informed consent signatures from parents or 
guardians of the children. The informed consent statements should be kept on file in 
a secure place. A.n example of an informed consent statement for a tutoring program 
is included in the Appendix. 

Data Collection 

Much of the useful quantitative data for tutoring programs may already be available 
from school records. Most schools keep records of daily attendance, retention, grades, 
and discipline referrals. Many schools will even be able lo access this information for 
a specific list of students who might participate in a tutoring program. The school 
counselor would probably be the first contact to make to acquire the information 
described above. 
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Remember to remove all names from materials and replace them with code 
numbers as soon as possible. This is necessary to protect the confidentiality of the 
students. Forms ancVor charts help condense collected data before entering the data 
into the computer. Samples of fonns which may be used as presented or adapted for 
use are inchjded in the Appendix. 

Schools may not compute a grade point average for elementary and middle school 
students. Directions for making this computation are included in the Appendix. 

instruments 

Some information will need to be obtained through surveys of participants, parents, 
teachers, and tutors. In this case, specific survey instruments may need to be adapted 
or developed. Examples of survey instruments are included in the Appendbc. 

Evaluators should plan enough time when ordering or developing needed assessment 
instruments to field-test the instruments. To field-test the instruments, the evaluator 
should ask a group of persons similar to those participating in the evaluation to 
complete the instuments. Revisions of the instruments should be based on the 
responses to this field test and on interviews with several of the field test respondents. 

A quantitative evaluation may require using standardized or other instruments to 
assess participants. The type of instrument needed will depend upon what will be 
evaluated. Depending on the goals and objectives of the specific program, some 
tutoring programs may use achievement tests while others may decide to evaluate self- 
esteem. However, the instruments must be selected or developed before they can be 
given to the participants in the evaluation. The following factors need to be considered 
when selecting an assessment instrument: 

• validity, 

• reliability, 

• cost, 

• scoring and administration procedures of the instruments, 

• age group for which the instrument is intended, 

• amount of time that it will take in order to locate and order thc uistnunent, 

• if the instrument truly measures what the program is attempting to measure, and 

• any special considerations of the population being tested. 

Since many tutoring programs identify increased self-esteem as a program 
objective, measuring the self-esteem of students involved in tutoring programs 
is a corrunon component of many evaluations conducted of tutoring programs. One 
of the most effective ways to measure self-esteem is to ask parents, teachers, and 
the child to rate the child*s self-esteem. 
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Examples: 

For parents: On a scale of 1-4 with 4 being highest, how would you rate 
your child's self-esteem? 

For students: What kind of learner are you? 

a very good learner 

pretty good learner 

not so good learner 

very bad learner 

In addition, there are quite a number of assessment instruments which have been 
developed to measure self-esteem. A section describing self-esteem tests that are 
available for elementary and high school students has been included in the Appendix. 
The list does not contain every existing self-esteem test or inventory, but it does 
provide an overview of what is available. The list was adapted from tests indexed in 
Tests in Print III (Mitchell, 1983). Although an assessment of self-esteem by one of 
the instuments described in the Appendix may be desirable, obtaining a rating by the 
child's parent(s) and/or the child's teacher may be more feasible, A parentis rating 
has the highest correlation with assessed self-esteem scores of children, followed by 
ratings of teachers. 

Assessment Conditions 

In addition to selecting the appropriate assessment instruments, evaluators must 
consider the conditions under which assessments will be made. Conditions under 
which participants are assessed should be as uniform as possible. The day and times 
that the instruments will be administered to participants may also have an influence. 
For example, giving participants a test on Halloween would probably not be a good 
idea because many participants have difficulty concentrating on this day. Other 
important conditions to consider include a quiet room, making sure participants have 
the appropriate materials to complete the assessment, and ensuring that the temperature 
of the room is adequate for the needs of those being assessed. Reading questions to 
all participants ensures that no participant is handicapped by not being able to read. 
A special answer sheet with larger print may also be helpful for some younger 
children. 

The test manual will list additional conditions to ensure the validity of the test 
being administered. It is a good idea to hold a training session for all of those who wiU 
be administering a test so they will be certain of the conditions necessary for giving 
the test. 

Scheduling Data Collection 

After developing or designing the assessment instruments, collect the data. A 
schedule for data collection will be helpful in planning for the method of collection 
and for organization of information that is collected. Evaluations of a tutoring 
program may require giving a test before the treatment starts (pre) and after the 
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treatment (post) to see if a significant difference has occurred. It is desirable to admin- 
ister the test before the trcaiment begins, but it is acceptable to administer the test after 
the treatment has begun as long as it is soon after the treatment has begun. Likewise, 
it is desirable to administer the posttest right before the treatment is scheduled to 
stop or right after the treatment has stopped. If the treatment happens to be an ongoing 
one, the evaluators must decide at what intervals the students will be assessed. 



Data Analysis 

There are different methods available for analyzing the data collected from 
evaluations. Most tutoring programs may need only a calculator to compute averages 
or percentages. If available,computcrs and commercial software packages may assist 
with the analysis. Software packages provide statistical analysis as well as visual 
information such as pie graphs and line graphs. For example, spreadsheets provide 
for easy data entry and allow evaluators to graph data and do statistical calculations. 
Spreadsheet programs can also incorporate data created by other programs and send 
data to other programs. Other statistical software packages require statistical 
understanding and programming skills. However, some of the programs are menu- 
driven and easy to use. The Appendix contains a representative list of names and 
addresses of companies that have either spread sheet programs or statistical software 
programs. 
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Reporting the Evaluation Findings 



The most common way of reporting the evaluation Tindings is through the use of 
both narrative and visual information. The main puipose of the report should be to 
organize all the gathered information, data analysis, and findings for use in making 
recommendations. The final report also needs to be comprehensive and written in a 
simple format so that audiences can easily obtain from it the information they need. 

The evaluation findings should be prepared in both written and oral formats. The 
oral format is particularly important for showing worthiness to obtain additional 
funding and for recruiting volunteers. An oral presentation should be prepared to 
include photographs, videotapes, slioes, graphs, and excerpLs from journals. A well- 
balanced presentation will reach both the mind and the heart of the audience. 

A suggested format for organizing the evaluation report (Indian Education Update, 
1991) is presented below. 

A. Introduction 

The introduction should give an overview of what will be presented in 
the report and the purposes of the evaluation. 

B. Description 

A description of the program and its goals and objectives should be 
discussed in this section. 

C. Procedures 

This section should give a description of the evaluation methods used in 
measuring the objectives of the program. Specifically, this section should 
include a discussion of the sources of data used, the methods used for 
collecting the data, the individuals involved in collecting the data, the 
validity and reliability of gathered data, and the accuracy of information 
obtained. 

D. Findings 

A presentation of the analysis of all the data gathered is the main focus of 
this section. Graphs and tables are often an excellent way to present and 
clarify the information gathered from the evaluation. A narrative section 
should describe the data that was collected, the results of the analysis, and 
the extent to which the objectives of the program were met. 

E. Summary of Findings and Recommendations 

The final section of the evaluation report is a summary of the evaluation 
results regarding which objectives were met and which were not. 
Recommendations for improving the program and for attaining unmet 
objectives should be included. This section might also contain suggestions 
for additional program objectives based upon the evaluation findings. 

Once the evaluation has provided data and information about the program, decisions 
can be made about changes in the program. Evaluation does not always show a need 
for change, but most evaluations do uncover speci fic areas that could use improvement. 
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Before the changes are made, the reasons why and the data supporting the need for a 
change should be presented to those involved in making decisions about changes. 
Based upon evaluation findings, rewrite goals and objectives to address the areas in 
which the program needs improvement. Set up additional methods for evaluating the 
changes that are made in the program. 

Example: 

The results of one tutoring program evaluation showed the overall reading 
grades of students improved, but the overall math grades went down. Tutors 
had been given more training in helping students improve their reading than in 
improving their math. Additionally, tutors were not aware of the need to review 
basic math facts with their tutees. Because of this finding, the training for the 
tutors was changed and the emphasis given to math and reading was equalized. 
Additional objectives were written to integrate more math into the tutoring 
sessions. 



Discipline Referrals 

3 T 



Graphs and tables are often an excellent 
way to present and clarify the information 
gathered from the evaluation. 
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Plan for Future Evaluation 



The next step is to plan for future evaluation. Evaluating educational programs 
should be an ongoing process, both formally and informally. Conduct a critique of the 
evaluation, particularly its strengths and weaknesses. Critiques often help pinpoint 
additional areas that need to be emphasized. When critiquing an evaluation the 
following questions are helpful: 

Did the evaluation gather all the data necessary? 

Did the evaluation look at all the variables needed to make decisions about 
the program? 

Was the data collected in an organized and ethical manner? 

Was the analysis of the data sufficient for the needs of the program? 

Did the evaluation provide enough information to allow the program to grow? 

Did the evaluation provide information to detennine if goals and objectives 
were met? 

Did the evaluation provide a framework on which to base additional 
evaluations? 
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PART VII 



Conclusion 



The advantages of ongoing evaluation of an educational program far outweigh the 
disadvantages. In addition to providing feedback for improving the program, 
evaluation also provides programs with opportunities to show how they are achieving 
their goals, what they are doing to improve the lives of the students they are serving, 
what they are doing for the community, and why they are worthy of receiving 
additional funding and the contributions of volunteers. 

A combination of both qualitative and quantitative research methods is usually the 
best approach because both methods uncover useful infonnation. Quantitative 
methods present numerical data. Qualitative methods, however, present richer 
information about the program and the people involved in it and try to represent the 
viewpoint of the participants. Both methods are important in order to get a holistic 
view of the program. 

The following table presents a summary of the minimal essential components of a 
tutoring program evaluation. It also distinguishes those components which would be 
easy and more difficult to add. 



Basic & Essential 


Easy to Add 


More Difficult to Add 


Tutoring Attendance 


School Attendance 


School Attitude 


Logs 


Records 


Survey 


Grades 


Discipline Referrals 


Specific Skill 






Assessments 


Homework Behavior 


Self-Esteem Ratings 








Assessment of Tutor/ 


Scheduled observation 


Journal Entries of 


Tutee Relationship 


of interactions between 


Tutor/Iutee 




Tutor/Tutee 




Questionnaires & 




Focus group of Tutors 


Surveys 


Anecdotal records & 






quotes of Tutor, Tutee, 


Videos 




Parents & Teachers 








Interviews 





Examples of 
participant's work 



Pictures 
Slides 
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SELF-ESTEEM TESTS 



Coopersmith Self-Esteem Inventories 

Stanley Coopersmith Consulting Psychologists Press, Inc. (1981) 

There are two versions of this test-lhe school form and the adult form. The school 
form is intended forages 8-15 while the adult form is forages 16 and above. Scores 
derived from the test include: general self-subscale score, social self-peers subscale 
score, home-parents subscale score, school-academic subscale score, total self-score, 
and the lie scale score. 

Culture-Free Self Esteem Inventories for Children and Adults 

James Battle; Special Child Publications (1981) 

There arc three different forms of this test. Form A, that contains sixty items, is 
for grades 3-9 and yields six scores: general, social/peer related, academics/school 
related , parents/home related, lie, and total score. Form B contains thirty items derived 
from form A. The adult form. Form AD, contains forty items and yields five scores: 
general, social, personal, lie, and a total score. 

Dimensions of Self-Concept 

William B. Michael and Robert A. Smith. Los .Angeles Unified School District (1 977-78) 
The first level of this testis for grades 4-6, and the second level is for grades 7-1 2. 
It is a self-report instrument that yields five factor scales: level of aspiration, anxiety, 
academic interest and satisfaction, leadership and initiative, and identification vs. 
alienation. 

Inferred Self-Concept Scale 

E.L. McDaniel, Western Psychological Services (1969-73). 

This test intended for grades 1-6 involves raungs by teachers and counselors. 

The Measurement of Self-Concept in Kindergarten Children 

Lucienne Y. Lcvinc and J. Clayton Lafferty; Research Concepts (1978) 

This test for kindergarten children relies on a projective drawing technique that 
produces three scores: self-concept, non-self-concept, and discrepancy. The last 
forms of this test available were experimental ones. 

Measures of Self-Concept 
Instructional Objectives Exchange (1972) 

This criterion-referenced test utilizes direct self-report, inferential self- report, and 
observation. Three levels are available: grades K-3,grades4-6, and grades 7-12. The 
primary version (grades K-3) contains ten objectives: self-appraisal (has five 
objectives: comprehensive, peer, school, family, and general), television actors, the 
class play, parental approval, work posting, and perceived approval situation. 

The intermediate version (grades 4-6) contains eleven objectives, self-appraisal 
(same as the primary version), what would you do, parental approval, the class play, 
how about you, work posting, and perceived approval situation. 




Self-Esteem Tests 



The secondary version (grades 7- 1 2) is made up of nine objectives: self-appraisal 
(same as above), what would you do, word choice, for all I know, and perceived 
approval situation. 

The Piers-Harris Children's Self Concept Scale (The Way I Feel About Myself) 
Ellen V. Piers and Dale B. Harris; Counselor Recordings And Tests (1969) 

Primary Self-Concept Inventory 

Douglas G. Muller and Robert Leonetti; Teaching Resources Corporation (1973-74) 
This test measures scl f-conccpt relevant to school success for grades kindergarten 
to grade six. Ten scores arc derived from the test: personal self (physical size, 
emotional state, total), social self (peer acceptance, helpfulness, total), intellectual self 
(success, student self, total), and a total score. 

Self-Concept Adjective Checklist 
Alan J. Politte; Psychologists and Educators, Inc. (1971) 
This test is intended for grades kindergarten to 8. 

Self-Concept and Motivation Inventory: What Face Would You Wear? 

George A. Farrah, Norman J. Milchus, and William Reitz; Person-O-Metrics, Inc. 
(1967-77) 

There arc three versions of this lest: the preschool/kindergarten form, the early 
elementary form, and the later elementary form. 

The preschool/kindergarten form isforages4-kindergarten. Three or fourscores 
can be derived: goal and achievement needs, achievement investment, self-concept, 
and an optional total score. 

The early elementary form for grades 1-3 produces fouror five scores: motivation 
(goal and achievement needs, achievement investment), self-concept (role expectation, 
self-adequacy) and an optional total score. 

The later elementary form is for grades 3-6 and produces 4- 1 1 scores: the same 
part scores as for Eariy Elementary Form plus six optional scores of support climate 
scores (parents, teachers, peers and siblings, academic self, academic activity climate, 
school climate, and an optional total score. 

The secondary fomi intended for grades 7-12 produces 4-20 scores: same part 
scores as for Eariy Elementary Form plus sixteen optional scores, sources of support 
climate (parents, teachers, peers, academic self, physical and social self, adults and 
counselors, academic activity climate, school climate), immediate-intrinsic orientation 
(evaluated competition, tasks and projects, discovery and creativity skills), fulfillment 
orientation (aspiration, cooperation and conformity, responsibility, acceptance and 
praise). 
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Self -Concept as a Learner Scale 
Waller B. Waeljen, the Author (1967-72) 

This test that is intended for grades 4-12 produces five scores: motivation, task 
orientation, problem solving, class membership, and a total score. No manual is 
available for this test. 

Seif-Esteem Questionnaire 

James K. Hoffmeister, Test Analysis and Development Corporation (1971-76) 

This test produces a self-esteem score and a self-other satisfaction score; it is 
intended for ages nine and over. 

SELF (Self-Concept Evaluation of Location Form) 

Richard E. Carney, Gil Spielberg, and Clifford W. Weedman; Carney, Weedman and 
Associates (1978-80) 

This test for adolescents and adults contains fifteen scores: total positive self- 
concept, evaluation factor, potency factor, activity factor, self-concept inconsistency, 
between factor inconsistency, within factor inconsistency, inconsistency F ratio, total 
self-ideal self-incongruence, evaluation incongruence, potency incongruence, activity 
incongruence, between factors incongruence, within factor incongruence, and 
incongruence F ratio. 

Tennessee Self Concept Scale 

William H. Fitts, Counselor Recordings and Tests (1964-65) 

There are two forms of this test intended for ages twelve and over. 
The counseling form has 14 profiled scores: self-criticism, nine self-esteem scores 
(identity, self-satisfaction, behavior, physical self, moral-ethical self, personal self, 
family self, social self, and total), 3 variability of response scores (variation across the 
first 3 self-esteem scores, variation across the last 5 self-esteem scores, and total), and 
a distribution score. 

The clinical and research form contains 29 profiled scores. The .vcores from the 
counseling form and then fifteen additional scores: response bias, net conflict, total 
conflict, 6 empirical scales (defensive positive, general maladjustment, psychosis, 
personality disorder, neurosis, personality integration), deviant signs, and 5 scores 
consisting of counts of each type of response made. 
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SPREADSHEET PROGRAMS 



Boeing Calc 
Company: 



Boeing Computer Services 
Softwarc and Education Products Group 
PO Box 2346, M,S, 
Seattle, WA 98124-0346 



Quattro 

Company: 



Borland International 
4585 Scotts Valley Drive 
Scotts Valley, CA 95066 



Lotus 1-2-3 

Company: 



Math Plan 

Company: 



Lotus Development Corporation 
55 Cambridge Parkway 
Cambridge, MA 02142 



Word Perfect Corporation 
288 West Center 
Oren, UT 84057 



SuperCalc 3 

Company: 



Super Calc 4 

Company: 



Computer Associates 
2195 Fortune Drive 
San Jose, C A 95131 



Computer Associates 
2195 Fortune Drive 
San Jose, CA 95131 



Multiplan 

Company: 



Microsoft Corporation 

1601 1 N.E. 36th Way, Box 97017 

Redmond, WA 98073 



VP-Planner 

Company: 



Paperback Software International 
2830 Ninth Street 
Berkeley, CA 94710 



PFS Professional Plan 

Company: Software Publishing Corporation 
1901 Landings Drive 
POBox 7210 

Mountain View, CA 94039 



^This material was adapted from information included in the Appendix of Principles of Educadonal and Psychological 
Measurement and Evaluation by Gilbert Sax. 



STATISTICAL SOFTWARE PACKAGES 



ABstat 
Company: 



BMPD 

Company: 



CRUNCH 

Company: 



Anderson-Bell Corp. 
South Pine Drive, Suite 411 
Parker, CO 80134 



BMPD Statistical Software, Inc. 
1440 Sepulveda Bh J., Suite 316 
Los Angeles, CA 90025 



Crunch Software 

5335 College Avenue, #27 

Oakland, C A 94618 



SAS 

Company: 



SAS Institute, Inc. 
Box 8000, SAS Circle 
Cary,NC 27511 



SPSSx and SPSS PC+ 

Company: SPSS, Inc. 

444 N. Michigan Avenue 
Chicago, IL 60611 

Stat 1-A Statistical Toolbox 

Company: Sugar Mill Software Corp. 
1180 Kika Place 
Kailua, Hawaii 96734 



CSS-Complete Statistical System 

Company: StatSoft 

2832 East 10th St., #4 
Tulsa, OK 74104 



Dyna-Stat 
Company: 



Microstat II 
Company: 



Minitab 

Company: 



Dynamic Microsystems, Inc. 
13003 Buccaneer Road 
Silver Springs, MD 20904 



Ecosoft, Inc. 

6413 North College Avenue 
Indianapolis, IN 46220 



Minitab, Inc. 

3081 Enterprise Drive 

State College, PA 16801 



NWA Statpak 

Company: Northwest Analytical, Inc. 

520 N.W. Davis Street 
PorUand, OR 97209 

PC Statistician 

Company: Human Systems Dynamics 
9010 Reseda Road, Suite 222 
NorthridgcCA 91324 



STATA 

Company: 



Stagraphics 

Company: 



StatPac Gold 

Company: 



StatPIan III 

Company: 



Statpro 

Company: 



Systat 

Company: 



Computing Resource Center 
10801 National Blvd., 3rd Hoor 
Los Angeles, CA 90064 



STSC, Inc. 

2115 E. Jefferson Street 
Rockvillle, MD 20852 



Walonick Associates 

6500 Nicollet Avenue South 

Minneapolis, MN 55423 



The Futures Group 
76 Eastern Blvd. 
Glastonbury, CN 06033 



Pcnton Software, Inc. 

420 Lexington Avenue, Suite 2846 

New York, NY 10017 



Systat, Inc. 

2902 Central Street 

Evanston,IL 60201 
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CHECKLIST OF STEPS TO GUIDE THE EVALUATION PROCESS 



Determine evaluation goals. 

discuss goals and objectives of program 

determine what questions the evaluation 

is attempting to answer 
establish goals for evaluation 

Develop an evaluation plan and outline the 
requircments of the evaluation, 

determine student identification variables 

determine measurable variables to be 

examined 

detennine what documentation is needed 

to answer each evaluation question 

determine who and how the data will be 

collected 

Determine how the evaluation will be structured 
and select the students that will participate in the 
evaluation. 

select the qualitative methods which will 

be used 

select the design of the evaluation 

select students that will participate in the 

evaluation 

Secure permission for students to participate in 
the evaluation. 

develop forms for parents to sign 

collect signed forms and verify that all 

parents have given their consent 

Determine what questions will be answered with 
interviews or focus groups, when the interviews 
or focus groups will take place and who will be 
interviewed or participate in the focus group. 

Select or develop the needed assessment 
instruments. 

research assessment instruments that are 

available for your puipose 
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select or develop the assessment 

instruments that will be used 

develop a timetable for when pre- and 

posttests will be administered 

plan for testing conditions and for any 

special provisions 

Collect the data. 

Develop or select forms that will be 

used to collect data 
assign codes to students and remove 

all names from documents 
gather qualitative and quantitative data 

Analyze the evaluation data. 

determine how data will be analyzed 

analyze both quantitative and 

qualitative data 

Report the evaluation findings, 
prepare a narrative report containing 

the results of the evaluations 
prepare visual information such as 

graphs, etc. 
analyze both quantitative and 

qualitative data 

Apply evaluation findings. 

discuss evaluation results 

formulate plans for necessary 

program changes 

Critique evaluation and make plans for future 
evaluation. 

determine strengths and weaknesses 

of evaluation 
map out plans for further evaluation 

. Disseminate the results of the evaluation. 
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TESTING CONDITIONS CHECKLIST 



nx)m temperature and room arrangements are comfortable for students 
test administrators arc knowledgeable about the test they are administering 

test administrators are aware of any special conditions for the testing or special conditions of students 
necessary testing materials are provided in the rooms 

special needs of testers have been considered and accommodations have been made for any of them 

no special events are planned for the days on which testing is scheduled 

students have been given the necessary information they need to effectively take t>.c test 

the pre- and posttesting dates were chosen in consideration of the program beginning and termination dates 
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PARENTS SURVEY FOR TUTORING PROGRAM EVALUATION 

1 . What three words would you use to describe the tutoring program in which your child is involved? 



2. What kinds of changes, if any, have you seen in your child since s/he started attending tutoring? 



3. What is the thing that you like the most about the tutoring program? 



4. WTiat is the thing you like the least about the tutoring program? 



5, Please check the areas in which you feel the tutoring has helped your child. 
Grades 

How your child feels about himself (self-esteem) 

Behavior at school 

Behavior at home 

Attendance 

Attitude towards school 

Other - Please Explain 

6. Does your child regularly talk about the tutor or the tutoring sessions? 
Yes No 

If yes, what are some of the things s/he discusses with you about the tutoring sessions? 



7. Please list and describe any comments and changes that you have for improving the tutoring program. 



Thank you for taking the time to complete this survey. 
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SURVEY FOR TUTORS 



1 . How do you think the tutoring program has helped the student you tutor? 



2. Please list and describe any changes you have noted in the student since the beginning of the tutoring program. 



3. What kinds of things do you do with your student during a regular tutoring session? 



4. Whc ' nds of things do you discuss/talk about with your student? 



5. How is the tutoring different than you expected? 



6. What do you think makes you a good tutoi? 



7. What is the toughest thing about being a tutor? 



8. How does tutoring help you? 



9. If you could change one thing about your student, what would it be? 



10. Please list and describe any comments and suggestions you might have for improving the tutoring program. 



Thank you for taking the time to complete this survey. 
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STUDENT SURVEY 



1. Do you like coming to tutoring? Yes No 

2. What are some of the things you do at tutoring? 

3. What do you like best about tutoring? 

4. What do you talk about with your tutor? 



5. Why do you come to tutoring? 

6. Check the word that describes how you feel about school. 

Fun Easy Boring Hard 

7. How does your tutor help you? 



8. Check the word that describes how much help you feel your tutor is giving you. 
a lot some a little none 

9. What would make tutoring better for you? 

10. If you could change one thing about your tutor, what would that be? 



Thank you for taking the time to aiiswer the questions. 
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TEACHER'S SURVEY 



1 . Why do you recommend students for the tutoring program? 

p oor grades 
^low self-esteem 

p oor attendance 

^discipline problems 

^lack of support from home 

needs help with homework 

^other (list) 



2. Have you seen improvement in your students* performance in any of the following areas? 

grades 

^discipline 

^behavior 

attendance 

self-esteem 

attitude towards school 

^homework completion 

3. In what way has the tutoring program been most helpful for the students you teach? 



4. How could the tutoring program be improved? 



5. Specifically, in what ways has the tutoring program helped you? 
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PARENTS INFORMED CONSENT FORM 



You are invited to participate in the evaluation of the ^Tutoring Program. The purpose of this 

evaluation is to determine how well the Tutoring Program is woiidng from the points of view of 

parents and children. The final goal is to help improve the tutoring services delivered through the 

Tutoring Program. If you agree to participate, you and your child will be asked to answer questions about the 
program. We have attached the questions you will be asked and the questions your child will be asked. Each parent 
and each child will be interviewed separately by a designated staff person from the program. The interviewer will 
be instructed to keep all of your answers confidential. Your participation in this evaluation is entirely voluntary. 
You have the right to withdraw your participation at any time up to the point it is no longer possible to identify the 
data you have provided to the evaluation. 

PROCEDURE: You will be asked to complete the attached questionnaire and return it to the project director. 

RISKS: There are no physical risks involved with this evaluation. If confi'ientiality is not preserved, your child's 
relationship with the tutor could become strained. To minimize this risk we have asked that neither you nor the 
person conducting the interview with you or with your child share information or discuss the information you 
provide. In addition, complete anonymity will be maintained in the evaluation report. 

EXCLUSION CRITERIA: Participation is determined by randomized selection. Participation is voluntary. As 
such, there are no exclusion criteria. 

BENEFITS: Participants will contribute to the improvement of services offered through the 
Tutoring program. 

COSTS: There will be no costs to participate in the evaluation. 

WITHDRAWAL: Participation in this evaluation is entirely voluntary. You may refuse to participate or you may 
withdraw from the evaluation at any time. This will involve no penalty to you or to your child. 

CONFIDENTIALITY: Your identity and sensitive information will be maintained as confidential by the 
safeguards set up within the evaluation. The interviewer will be instructed not to discuss the answers provided by 
the children and parents with anyone. 

QUESTIONS: For more information concerning this evaluation, you may contact the program director or the 
evaluator of the project: 

at 

(Project Director) (telephone number) 

or at . 

(Project Evaluator) (telephone number) 



Parent's initials Date 
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Parent's Informed Consent Form 



CONSENT: 

I have been given an opportunity to ask questions about this evaluation; answers to such questions (if any) have 
been satisfactory. The information in the evaluation records will be kept confidential. My child's identity and my 
identity will be anonymous. 

I agree to participate in the evaluation. I understand that the information obtained wiU be used for this evaluation 
only, that it will be treated confidentially and that no names or school names will be used in the final report. 

1. I agree to complete the questionnaire for the Evaluation of Tutoring Program. 

2. I understand that the information obtained will be used for this evaluation only, that it will be treated 
confidentially and anonymously, and no identifying information will be used in the final report. 

3. I agree to participate in a focus group discussion, if selected. 

4. I agree to be interviewed, if selected. 

Thank you for your help and cooperation. If you would like a copy of the evaluation report, please indicate below. 



Parent's Signature Date 



Witness Signature Date 



44 



51 



HOW TO COMPUTE A GRADE POINT AVERAGE 



To compute a grade point average for a student, use the following 
values for each letter grade: 

A = 4.0 
B = 3.0 
C = 2.0 
D = 1.0 
F = 0.0 

Add all of the numbers and divide the total obtained by the number of 
grades earned by the student. 

The resulting figure is the student's grade point average. 

For example, suppose that Jason earns the following grades: 



Reading 


C = 


2 


Math 


D+ = 


1 


Social Studies 


C = 


2 


Science 


B- = 


3 


Total 




8 



What is Jason's grade point average? Divide 8 by 4. The answer is 2.0. 
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Evaluation Guide for Tutoring Programs® is a part of a series of dropout prevention research reports 
published by the National Dropout Prevention Center. Additional copies may be ordered. The Center 
has produced a variety of other products which can be helpful to those who work with 
youth in at-risk situations. 

To obtain a complete list of publications and prices call or write: 

Publications Department 

The National Dropout Prevention Center 

205 Martin Street 

Clemson University 

Box 345111 

Clemson, South Carolina 29634-5 111 50 
(803) 656-2599 
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